Idempotent Expansions for Continuous-Time Stochastic Control

It is now well-known that many classes of deterministic control problems may be solved by max-plus or minplus (more generally, idempotent) numerical methods. These methods include max-plus basis-expansion approaches [1], [2], [6], [9], as well as the more recently developed curseof-dimensionality-free methods [9], [14]. It has recently been discovered that idempotent methods are applicable to stochastic control and games. The methods are related to the above curse-of-dimensionality-free methods for deterministic control. In particular, a min-plus based method was found for stochastic control problems [10], [15], and a min-max method was discovered for games [11]. The first such methods for stochastic control were developed only for discrete-time problems. The key tools enabling their development were the idempotent distributive property and the fact that certain solution forms are retained through application of the semigroup operator (i.e., the dynamic programming principle operator). In particular, under certain conditions, pointwise minima of affine and quadratic forms pass through this operator. As the operator contains an expectation component, this requires application of the idempotent distributive property. In the case of finite sums and products, this property looks like our standardalgebra distributive property; in the infinitesimal case, it is familiar to control theorists through notions of strategies, non-anticipative mappings and/or progressively measurable controls. Using this technology, the value function can be propagated backwards with a representation as a pointwise minimum of quadratic or affine forms. Here, we will remove the severe restriction to discrete-time problems. This extension requires overcoming significant technical hurdles. First, note that as these methods are related to the max-plus curse-of-dimensionality-free methods of deterministic control, there will be a discretization over time, but not over space. We will first define a parameterized set of operators, approximating the dynamic programming operator. We obtain the solutions to the problem of backward propagation by repeated application of the approximating operators. These solutions are parameterized by the timediscretization step size. Using techniques from the theory of viscosity solutions, we show that the solutions converge to the viscosity solution of the Hamilton-Jacobi-Bellman partial differential equation (HJB PDE) associated with the original problem. The problem is now reduced to backward propagation by these approximating operators. The min-plus distributive property is employed. A generalization of this distributive property, applicable to continuum versions will be obtained. This will allow interchange of expectation over normal random variables (and other random variables with range in IR) with infimum operators. At each time-step, the solution will be represented as an infimum over a set of quadratic forms. Use of the min-plus distributive property will allow us to maintain that solution form as one propagates backward in time. Backward propagation is reduced to simple standardsense linear algebraic operations for the coefficients in the representation. We also demonstrate that the assumptions on the representation which allow one to propagate backward one step are inherited by the representation at the next step. The difficulty with the approach is an extreme curse-ofcomplexity, wherein the number of terms in the min-plus expansion grows very rapidly as one propagates. The complexity growth will be attenuated via projection onto a lower dimensional min-plus subspace at each time step. At each step, one desires to project onto the optimal subspace relative to the solution approximation. That is, the subspace is not set a priori. In the discrete-time case, it has been demonstrated that for some problem classes, this approach is substantially superior to grid-based methods. Simple numerical examples with continuous-time dynamics will be examined with this new approach.